Ratings are Overrated!

نویسندگان

Georgios N. Yannakakis

Héctor Perez Martínez

چکیده

Are ratings of any use in human–computer interaction and user studies at large? If ratings are of limited use, is there a better alternative for quantitative subjective assessment? Beyond the intrinsic shortcomings of human reporting, there are a number of supplementary limitations and fundamental methodological flaws associated with rating-based questionnaires – i.e., questionnaires that ask participants to rate their level of agreement with a given statement, such as a Likert item. While the effect of these pitfalls has been largely downplayed, recent findings from diverse areas of study question the reliability of using ratings. Rank-based questionnaires – i.e., questionnaires that ask participants to rank two or more options – appear as the evident alternative that not only eliminates the core limitations of ratings but also simplifies the use of sound methodologies that yield more reliable models of the underlying reported construct: user emotion, preference, or opinion. This paper solicits recent findings from various disciplines interlinked with psychometrics and offers a quick guide for the use, processing, and analysis of rankbased questionnaires for the unique advantages they offer. The paper challenges the traditional state-of-practice in human–computer interaction and psychometrics directly contributing toward a paradigm shift in subjective reporting.

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

A Personalized Recommendation System for Yelp Users Name SUNetID

Yelp, a platform that consists of various business information, allows the users to provide star ratings and text reviews for businesses they visited before. This provides insights for other potential users. Particularly for restaurants, most users choose based on two different considerations, the overall ratings and the reviews posted by other users. However, there is a severe problem associat...

متن کامل

Self – Others Rating Discrepancy of Task and Contextual Performance

This research compared ratings of task performance and contextual performance from three different sources: self, peer, and supervisor. Participants were service industry employees in the service industries in Yogyakarta, Indonesia. A Sample of 146 employees and 40 supervisors from the service industries provided ratings of task performance and contextual performance. The results indicated th...

متن کامل

Is evidence-based medicine overrated in family medicine?: No.

YES. It might seem strange to purport that evidence-based medicine (EBM) is overrated. It has been described as a Copernican revolution in medicine, celebrated as one of the most important ideas of the 20th century, and considered the leading movement in modern clinical medicine. It is claimed to be the standard by which all medicine should be practised.1 Surely arguing against EBM is akin to s...

متن کامل

Analyzing the Impact of Credit Ratings on Firm Performance and Stock Returns: Evidence from Taiwan

The respective study covers three aspects; factors determining credit rating, credit rating impact on performance of entities and the relation between stock returns and credit rating. The study focuses on the firms listed in Taiwan Stock Exchange (TSE) of Taiwan. The empirical analysis uses the data of 50 firms rated by Taiwan Ratings Corporation (TRC) for the period 2010-2015. Two estimation t...

متن کامل